Skip to content

Eliminate Reliance on CI Platform Settings in generate_workflows.sh and Prioritize Hosts Configuration#4664

Open
AntonMFernando-NOAA wants to merge 26 commits intoNOAA-EMC:developfrom
AntonMFernando-NOAA:bug/generate_wf_hostfiles
Open

Eliminate Reliance on CI Platform Settings in generate_workflows.sh and Prioritize Hosts Configuration#4664
AntonMFernando-NOAA wants to merge 26 commits intoNOAA-EMC:developfrom
AntonMFernando-NOAA:bug/generate_wf_hostfiles

Conversation

@AntonMFernando-NOAA
Copy link
Contributor

@AntonMFernando-NOAA AntonMFernando-NOAA commented Mar 17, 2026 •
edited
Loading

Description

  • This PR resolves the issue where generate_workflows.sh depended on dev/ci/platforms and overrode dev/workflow/hosts settings (e.g., using ICSDIR_ROOT instead of BASE_IC). It restores precedence to the hosts configuration, removes the unnecessary reliance on CI settings for manual runs, and eliminates ICSDIR_ROOT from the codebase as it is redundant.

    Resolves generate_workflows uses CI settings instead of those in hosts file #4636

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? NO (If YES, please indicate to which system(s))
    • GFS
    • GEFS
    • SFS
    • GCAFS
  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

  • Run affected tests across all platforms up to the stage_ic job and verify that the BASE_IC variable is used correctly.

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@AntonMFernando-NOAA AntonMFernando-NOAA changed the title Eliminate Reliance on CI Platform Settings in generate_workflows.sh and Prioritize Hosts Configuration Eliminate Reliance on CI Platform Settings in generate_workflows.sh and Prioritize Hosts Configuration Mar 17, 2026
AntonMFernando-NOAA and others added 2 commits March 17, 2026 12:49
@AntonMFernando-NOAA @github-actions
Update dev/workflow/generate_workflows.sh
312e5af 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@AntonMFernando-NOAA
Add default argument to create_experiment test
6d719f1
AntonMFernando-NOAA and others added 3 commits March 17, 2026 19:08
@AntonMFernando-NOAA
rework BASE_IC
1b93ac4
@AntonMFernando-NOAA
update postamble
8569476
@AntonMFernando-NOAA @github-actions
Update ush/postamble.sh
c33efe7 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@AntonMFernando-NOAA AntonMFernando-NOAA self-assigned this Mar 17, 2026
@AntonMFernando-NOAA AntonMFernando-NOAA marked this pull request as ready for review March 19, 2026 18:19
@AntonMFernando-NOAA
Copy link
Contributor Author

AntonMFernando-NOAA commented Mar 19, 2026

@WalterKolczynski-NOAA @TerrenceMcGuinness-NOAA @DavidHuber-NOAA I tested most of the affected CI tests on hera using the updated generate_workflow.sh script. Let me know if you’re okay with these changes. If so, I’ll run a comprehensive CI test suite across all platforms to confirm everything works as expected. Also, please let me know if there are any additional tests I should include.

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

TerrenceMcGuinness-NOAA commented Mar 20, 2026 •
edited
Loading

This looks real good. Getting ride of the dependency of ICSDIR_ROOT as an environment variable for the CI pipeline codes for intake of the PR case yaml files was most helpful. I do see though this also needs to be panned out for the Functional CTest Jobs framework as well as we had that environment variable dependency forced into there as well. I'll propose we create follow up issue to test and workout what is needed there. I'll assign that one to myself if that's ok.

@AntonMFernando-NOAA
Copy link
Contributor Author

AntonMFernando-NOAA commented Mar 20, 2026

This looks real good. Getting ride of the dependency of ICSDIR_ROOT as an environment variable for the CI pipeline codes for intake of the PR case yaml files was most helpful. I do see though this also needs to be panned out for the Functional CTest Jobs framework as well as we had that environment variable dependency forced into there as well. I'll propose we create follow up issue to test and workout what is needed there. I'll assign that one to myself if that's ok.

@TerrenceMcGuinness-NOAA Thanks, Terry!

WalterKolczynski-NOAA
WalterKolczynski-NOAA previously approved these changes Mar 23, 2026
View reviewed changes
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know about numbering the steps in the comments because it becomes annoying to update if you add a step in the middle, but that's up to EMC. Changes look like they work, so if ctests and CI/CD all work, I'm happy.

@AntonMFernando-NOAA
Copy link
Contributor Author

AntonMFernando-NOAA commented Mar 23, 2026 •
edited
Loading

I don't know about numbering the steps in the comments because it becomes annoying to update if you add a step in the middle, but that's up to EMC. Changes look like they work, so if ctests and CI/CD all work, I'm happy.

@WalterKolczynski-NOAA Thanks for the review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA left review comments

@DavidHuber-NOAA DavidHuber-NOAA Awaiting requested review from DavidHuber-NOAA DavidHuber-NOAA is a code owner

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Awaiting requested review from TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA is a code owner

@TravisElless-NOAA TravisElless-NOAA Awaiting requested review from TravisElless-NOAA TravisElless-NOAA is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@AntonMFernando-NOAA AntonMFernando-NOAA

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

generate_workflows uses CI settings instead of those in hosts file

3 participants

@AntonMFernando-NOAA @TerrenceMcGuinness-NOAA @WalterKolczynski-NOAA